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A human-in-the-loop evaluation of the Operational Airspace Sectorization Integrated 
System (OASIS) was conducted in the Airspace Operations Laboratory at NASA Ames 
Research Center. OASIS is an advisory tool built on an Android touch tablet, designed to 
assist Federal Aviation Administration (FA A) En Route Area Supervisors in their planning 
of sector combine/split operations as well as opening/closing of radar associate control 
positions over the subsequent two hours. During the experiment, eight retired FAA 
personnel served as participants for a part-task evaluation of the OASIS user interface and 
the underlying mathematical algorithm that provided the advisories. There were three 
experimental conditions: Baseline, Computer Recommend Plan (CRP), and User Generated 
Plan (UGP). In the Baseline condition, participants were presented with four different traffic 
scenarios and were asked to generate their own sector configuration plan solutions without 
OASIS. In the CRP condition, they evaluated the multiple advisory solutions that were 
generated by OASIS. In the UGP condition, they modified the OASIS advisory solutions to 
make their own solutions with the support of the OASIS tool. The participants considered 
the OASIS advisory solutions at least as good as their own, suggesting that the underlying 
algorithm provided good solutions for the Area Supervisors. In the UGP condition, the 
participants could not improve on the OASIS advisories by further tweaking the solutions. 
Participants gave positive feedback on both the user interface and the algorithm solutions, 
including an excellent average rating above 90% on the tool usability scales. They also 
suggested various enhancements to be incorporated into the next tool development cycle. 
The development of OASIS is a major activity of the Dynamic Airspace Configuration 
(DAC) research focus area within the Airspace Systems Program. 
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I. Introduction 

Airspace management is a key element in guiding the safe and efficient operations of the National Airspace 
System (NAS). Airspace management requires allocating the appropriate airspace capacity to the predicted traffic 
demand in order to maximize the airspace utilization in the NAS. The current NAS architecture is reaching the limits 
of its ability to accommodate increases in traffic demand. In the foreseeable future, the traffic is predicted to 
increase substantially, creating an environment in which effective balancing of demand and capacity becomes a high 
priority in the Next Generation Air Transportation System (NextGen) timeframe . 1 At the National Aeronautics and 
Space Administration (NASA), a team of researchers have been exploring methods to efficiently allocate airspace 
capacity to the predicted traffic demand under Dynamic Airspace Configuration (DAC) project . 2 

In contrast to today’s operations in which the airspace is managed by splitting and combining a set of static 
sectors in response to the changes in the traffic demand, the researchers in DAC project have explored a variety of 
methods to find different sector geometries that could flexibly change over time in order to adapt to the real-time 
traffic demands. To that end, the formulation, development, and testing of different algorithms that can divide the 
airspace to optimally meet the traffic demand has been an area of concerted effort. A number of algorithms have 
been developed and refined as a part of DAC project, each with a differing approach to dynamic sectorization using 
different input metrics and goals to inform the calculations . 3 A direct comparison of the different airspace 
sectorization algorithms has shown that no one algorithm is superior to others. Rather, each algorithm makes 
different tradeoffs between competing metrics, such as a reduction of the aircraft delay at a cost of large airspace 
volume shifts during the reconfiguration or vice versa. A set of human-in-the-loop (HITE) simulations that tested a 
subset of the airspace sectorization algorithms also confirmed the tradeoff between the benefits and reconfiguration 
costs . 4,5 The algorithms that increased the capacity more than the other algorithms also resulted in higher airspace 
volume shifts during reconfigurations which in turn resulted in higher controller workload. 

Different algorithms are also geared towards different NextGen implementation timeframes, with some 
assuming technologies that are not expected to be available until very far in the future. In an effort to shift the 
research focus to airspace sectorization algorithms that could be integrated into near to mid-term NextGen 
environments, a tool prototype called Operational Airspace Sectorization Integrated System (OASIS) was developed 
and refined based on one of the algorithms that had the best potential for near to mid-term implementation. This 
algorithm developed by Bloem and his colleagues provided the operational benefits of better airspace sectorization 
without making drastic changes to the existing airspace infrastructure . 6 The following sections describe the OASIS 
tool and its operational context. 
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II. Background 

In air traffic control (ATC), an Air Route Traffic Control Center (called ARTCC or Center) is a facility 
responsible for controlling aircraft in a volume of airspace in the en route phase of the flight. The Center airspace is 
located at high altitudes, typically above 10,000 feet. A Center divides its airspace into several “areas of 
specialization” (also called Areas), generally consisting of six to nine sectors. Each sector is managed by at least one 
air traffic controller called the radar or R-side controller. As the traffic demand increases in a sector, a second 
controller called the radar associate or D-side controller is added to assist the R-side controller if the aircraft volume 
in the sector exceeds the workload capacity of a single controller. 

The traffic demand increases or decreases significantly as traffic flow between airports ebb and flow. Over a 
course of a day, traffic ramps up from early in the morning and ramps back down to a trickle late in the evening. As 
the traffic demand changes, the sectors are routinely combined or split to balance the airspace resources to the traffic 
demand in order to provide manageable but sufficient traffic demand in each sector. Airspace changes (e.g., sector 
consolidation or splitting) are accomplished by executing pre -defined solutions for changing the vertical and lateral 
boundaries between sectors. Some of the events that change the balance between resources and demand include 
changes in traffic volume, weather, special use airspaces, oceanic track change, and equipment outage. 7 

When a sector is combined from multiple sectors to a single sector, the process is initiated by a Front Line 
Manager (FLM; also called Area Supervisor or Supervisor). The procedure involves choosing one of the initial 
sectors to be active for the resulting combination and the other sectors to be inactive. The controller at the active 
sector assumes control of aircraft in the sectors designated to be inactive. Sector combination is commonly used 
during periods of light traffic such as night-time flow shifts. Figure 1 illustrates how the daytime sectors are 
combined in the night time operations to create larger sectors in the Kansas City Center airspace. The nighttime 
sectors are typically combined both laterally and vertically. 



Figure 1. Basic Kansas City ARTCC (ZKC) sector plan (left) and night-time consolidation of sectors (right) 


Conversely, splitting a previous combined sector into multiple sectors occurs when the traffic volume is 
predicted to exceed the capabilities of the sector. For example, during the peak traffic hours, there are sectors with 
many aircraft that fly along the same set of jet routes at different altitudes such that they exceed the capabilities of 
the sector being transited. Under these conditions, the sectors are often split vertically at a particular altitude to 
reduce the aircraft count per sector and ease the controller workload. 

As the traffic demand changes, the Area Supervisor manages the controller workload by combining/splitting 
sectors and/or adding/subtracting D-side controllers. For example, an Area Supervisor can combine adjacent sectors 
under the control of one R-side controller as the traffic load decreases. When the traffic load increases at a later 
time, the Supervisor can then choose to either add a D-side controller to the combined sector, or split the combined 
sector back into separate ones, each with an R-side controller. 

In today’s operations, the Area Supervisor makes these decisions without any tools. OASIS is an advisory tool 
that has been prototyped to assist Area Supervisors in their planning of sector combine/ split operations and 
opening/closing D-side control positions. The OASIS algorithm identifies airspace configuration options, available 
operating positions, and the timing of configuration changes for the next two to three hours of air traffic operations. 
It suggests a “best” airspace configuration to move from the current situation to an ideal end state while ensuring 
that air traffic is being efficiently and safely managed. This includes recommendations for the sector configurations 
and D-side assignments for each configuration change. The advisories also provide the timing of the sector 
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configuration changes that would minimize the impact on the controller task load during the changes. Figure 2 
summarizes the inputs that OASIS uses to make its recommendations/outputs. 



Figure 2. OASIS Inputs and Outputs 
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One of the key assumptions about OASIS and its usage is that OASIS provides advisories/recommendations to 
the Area Supervisor who makes the final decisions. The concept assumes that the OASIS algorithm can make 
intelligent recommendations on the sector configuration changes and D-side assignments based on the available 
position and predicted traffic information that the Supervisor might not have considered without the tool. However, 
the concept also recognizes that there are other factors not considered by OASIS that the Supervisor also needs to 
consider in making his/her sector configuration decisions. The goal of the OASIS tool is to work in a 
complementary fashion to the existing decision making process by the Area Supervisors. 


III. OASIS Tool Development 

The development of the OASIS tool began in the fall of 201 1. Although the underlying algorithm that provides 
the OASIS advisories had been under development for a number of years, a functional tool with an interface that 
users can access had yet to be created. Therefore, a core team of human factors experts, an interaction designer, and 
operational subject matter experts (SMEs) at NASA Ames Research Center was tasked with designing a user 
interface for the OASIS tool. The ATC SMEs assessed the feasibility and real-world application of the tool. Human 
factors researchers ensured the proper application of research techniques and provided team oversight. An 
interaction designer took the team through the interface invention and design process and a software developer was 
tasked with implementing the team’s design and coupling it with the underlying algorithm. Additional support 
research staff members were responsible for experimental setup, data collection and processing assistance. The tool 
development process is illustrated in Figure 3. 

The process consisted of a “Design, Test and Repeat” approach. In the design phase, the first steps were to 
gather all of the information from past meetings/papers related to the DAC project to define the high level goals of 
OASIS. In addition, other existing tools similar to OASIS were identified and reviewed to ensure that the OASIS 
tool would provide new capabilities. SMEs were solicited so as to better understand how the sector configuration 
task is performed in today’s operation. The goal was to ensure that OASIS would work in a complementary way 
with the Area Supervisor’s decision making process. 

Necessary features and functions of the tool were defined, cataloged, and prioritized. Use cases were developed 
to build various test scenarios and identify the operational contexts in which OASIS could be used by the Area 
Supervisors to improve their decision making process. Low fidelity sketches were created during ideation to refine 
the initial concept of operations, tool features, and functional requirements. A diagram was created to assess the 
logic flow of the architecture by applying various interface concepts step-by-step via screen transitions. A low 
fidelity wireframe, or interface flow, was created to express how the tool might function in finer detail. The team 
assessed each step of the design process via presentation of work product (design critique) until a medium fidelity 
mockup, or ’’click-through," was created. This click-through was then presented to a larger audience of researchers 
and SMEs for review and all feedback was put into the process anew. 
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Figure 3. OASIS Design Process 

In the test phase, several iterations of the design process were completed before a functional prototype of the 
OASIS tool was coded in Java on a portable tablet computer. The prototype was debugged, tested, and ultimately 
evaluated via a HITL simulation, accompanied by a user survey. The initial simulation provided valuable qualitative 
and quantitative measures that the team distilled, prioritized and fed back into the design process. 

A full description of the design and test phases of the OASIS tool development is outside the scope of this 
paper. However, following sections will describe the initial definition of information requirements, its usage, and the 
OASIS tool that resulted from the process. 

A. Initial Definition of Information Requirements, Tool Usage, and Form Factors 

At the beginning of the design phase, the team was given a set of initial deliverables and assumptions about the 
OASIS tool functionalities. The tool was to be based on an optimization algorithm that could output sectorization 
and D-side assignment advisories based on available personnel and the predicted traffic demand. The algorithm was 
envisioned to operate in the background, calculating new solutions in response to the changes in the predicted traffic 
loads and the available personnel. The tool interface needed to be designed to present one or more algorithm- 
generated sector configuration and D-side assignment solution. It also needed to present all of the “appropriate data” 
to the Area Supervisor to provide situational awareness of upcoming events that could affect sector reconfiguration. 
The ultimate decision as to whether the chosen airspace configuration solutions were appropriate and should be 
implemented was left to the supervisor. 

The workflow was 1) User inputs staffing schedule for a 2-3 hour timeframe; 2) OASIS allows user to interact 
with and evaluate advisories in order to select one of the algorithm-generated advisory solutions; and 3) User 
modifies one of the advised solution as s/he sees fit. An additional (though not strict) request was that the tool be run 
on a platform that allowed it to be portable. Possibilities included a convertible laptop with touch screen or various 
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tablet platforms. The team was to present the best approach for the intended user audience and make 
recommendations on their behalf. 

The SMEs on the team evaluated the tasks and information requirements for the Area Supervisors who are the 
intended users of the OASIS tool. The SMEs summarized user decision processes for the Area Supervisors as "Scan, 
Recognize, Assess, Plan, Execute" (Figure 4). SMEs also identified the existing tools and platforms that Supervisors 
relied on. These included Cru-ART for personnel related information, User Request Evaluation Tool (URET) for 
aircraft-related information, and the Monitor Alert Parameter (MAP) from Enhanced Traffic Management System 
(ETMS) for predicted traffic load, (Figure 4). During the scan, Supervisors could recognize potential problems, such 
as predicted traffic over maximum sector threshold, as shown from the ETMS data. S/he could assess the severity of 
the problem by “drilling down” on the data to gather further information. If the problem required an action, s/he 
could formulate and execute a plan to resolve the issue, such as splitting sectors and/or adding D -sides to the 
overloaded sectors. 




Radar Presentation 


Figure 4. Area Supervisor Decision Process 



After an initial assessment of the tasks and the potential OASIS tool usage, the team concluded that the Area 
Supervisors would likely be busy performing a variety of tasks and therefore were unlikely to interact with OASIS 
for more than three to four minutes at a time. This constraint was placed on the team and became a key variable. The 
team visited various air traffic facilities to observe users and better understand the environment in which the tool 
would perform. Lighting, noise, physical space, personnel layout, and other factors were noted. It was also 
determined through research that the look-ahead time of the algorithm should be constrained to two hours. This was 
based on the current look-ahead time horizon in the En Route air traffic facilities and the uncertainty of the 
underlying predicted traffic data. The two hour time constraint also coincided with the FAA recommendation that 
ETMS look-ahead time be constrained to 1.5 to 2.5 hours in length. The two hour time horizon was isolated as 
having impact on the layout of interface elements such as timelines and alerts. 

For the determination of the form factors of the OASIS tool, the team considered both laptop computers and 
tablets and decided on a light, tablet-based system that was not tethered to a keyboard and was easy to carry. The 
decision was based on various factors: minimizing the footprint of a new tool, a more enhanced user interface 
developed for the tablets, etc. In addition, the tablets had the flexibility to function as a portable wireless device in 
the future environments. However, the initial design assumed the tool to be wired since wireless devices are not 
permitted in current air traffic environments. 

Through initial evaluation of the different systems, it was determined that an Android tablet was capable of 
performing the algorithmic computation while also providing many pre -built user interface elements. Due to 
engineering and design challenges, it was deemed a priority to draw on the design patterns and pre -built interface 
components of Android 4.0 as much as possible. It was hoped that this would save valuable programming and 
component-design time as well as maintaining code portability should Android specifications require compliance to 
design standards laid out by Google. Many of the designs presented had to be carefully considered and researched 
for coding feasibility before implementation. 
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B. Development of Use Cases 

Once the team defined the initial requirements and the tool usage/functionalities, the next step was to develop the 
use cased to further define the tool usage in the operational context. The SMEs on the team discussed and arrived at 
five different use cases in which the OASIS tool might potentially aid the Area Supervisor in deciding on the best 
sector configurations and D-side assignments. Some of the use cases applied to sector configuration changes that 
occur daily, while others highlighted off-nominal cases in which the solutions might be difficult to find. Following 
are the five use cases that were identified: 

• Mid-shift Change - At the beginning of the mid-shift when the traffic volume goes down or at the end of the 
mid-shift when the traffic volume goes back up, the Area Supervisor plans and executes sector combinations 
to reduce the number of controllers when the traffic is reduced in the evening and sector splitting to add 
controllers when the traffic volume increases in the morning. 

• Staffing Change - During the course of the shift, the Area Supervisor plans and executes sector 
configurations to accommodate the release and return of controllers to/from collateral activities and details 
or to evaluate requests for leave. 

• Traffic Surge - During the day where there is high traffic volume, a traffic management initiative is 
instituted that will result in a traffic surge at some of the sectors in the area, which are currently combined. 
After being alerted, the Area Supervisor assesses the situation and splits the sectors, planning to recombine 
them when traffic permits prior to the end of the shift. 

• Weather-related Traffic Change - A weather forecast predicts a high probability of the blockage of routes in 
one or more of the sectors in the area. This forecast if accurate will result in the closure of the impacted 
sector and extra traffic demand on surrounding sectors. 

• Equipment Outage - One or more of the workstations and/or voice communication systems become 
unavailable due to a failure or a scheduled maintenance. The Area Supervisor plans and executes sector 
configurations to accommodate the predicted traffic demands without the unavailable equipments. 

Appendix A provides an example of the use case developed for the Mid-Shift Change scenario. A detailed step- 
by-step description of the trigger events that requires the sector configuration changes and how the Area Supervisor 
might use the OASIS tool in this context allowed the team to specify the details of the tool functionalities and user 
interface components needed to accomplish these tasks. 

C. Prototype of OASIS Tool 


General Interface Interactions 
Architecture 



Figure 5. Initial Sketches of the OASIS Screen Layout 
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An initial interface of the OASIS tool was created and iterated using low fidelity sketches and a diagram of the 
logic flow of the tool functionalities that were outlined in the use cases. A basic screen layout was designed to 
identify the main set of interface components and to place them on the tool in a consistent manner. Three main 
categories of interface components were initially identified as “Views”, “Modes”, and “Actions”. An initial layout 
that was considered is shown in Figure 5. This initial layout was modified and refined further for the actual 
prototype. 

The OASIS tool was designed to be used in one of three modes: Active Plan, Computer Recommended Plan 
(CRP) and User Generated Plan (UGP) mode. The Active Plan mode displayed a schedule of sector configuration 
changes and D-side assignments that were expected to be implemented in the next couple of hours. The CRP mode 
allowed the Area Supervisor to query and examine multiple algorithm solutions that the OASIS tool calculated. If 
s/he liked one of the solutions, s/he could use it to replace the current Active Plan. In the UGP mode, the Area 
Supervisor could inherit a plan from either the Active Plan or CRP and modify it to create a new plan that met 
his/her needs or preference. The UGP mode was intended to provide a mechanism to allow the Supervisor to modify 
algorithm generated advisories, thereby creating “hybrid” plans that could potentially be superior to both algorithm 
or human generated plans. In each of the modes, the user was able to access different views of the sector 
configuration, D-side assignments, the predicted traffic information, and other relevant data. Initially, the team 
considered showing the sector configurations and the D-side assignments in either a tabular format or in a format 
that shows the room / workstation configuration of the physical Area. In the original design, the predicted traffic 
data were to be accessed only on demand. However, when the initial sketches and the interaction flows of the 
different views were presented to a wider audience of SMEs, they provided a strong feedback that the interface 
needed to present the predicted traffic information along with the sector configurations and D-side assignments at 
the top pages of the interface. Therefore, the Views components of the interface were completely redesigned for the 
actual functional prototype in response to their feedback. 

After multiple iterations of the interface sketches, functional specifications and interaction flow diagrams, the 
actual functional prototype of the OASIS tool was built on an Android tablet (Figure 6). As discussed earlier, the 
tool had three modes (i.e. Active, CRP, and UGP) and the different modes were accessible by pressing the mode 
buttons at the bottom of the screen (see Figure 7 for a better screen shot). Within each mode, three graphical data 
presentation views were identified: Plan View, NAS Monitor View and one -minute Load Graph View. The user 
could switch between the three views by simply swiping the display either left or right. 



Figure 6. OASIS Tablet in Use (showing NAS Monitor View) 


Figure 7 illustrates the Plan View for the Active Plan (i.e., the sector configuration plan that is currently being 
executed). The Plan View gave the user an overview of the two-hour planned schedule of sector configuration 
changes and D-side assignments (top graph called Configuration Plan Graph) and the corresponding traffic loads 
(bottom graph called Sector Area Load Graph) on a single page. The top graph in the Plan View showed five sectors 
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in the Area (i.e., sectors 45, 46, 47, 48, and 49) on the vertical axis and the two-hour duration from when the plan 
was initiated to the end of the plan on the horizontal axis. The Coordinated Universal Time (UTC) was shown across 
the top of the graph, along with a “time gutter”, whenever there was a change in the sector configurations or D-side 
assignments. The number of horizontal bars next to any given sector number indicated the number of sectors that 
were combined and assigned to that particular sector. For example in Figure 7, it shows that from the time 10:30 
UTC (labeled “1030Z”) to 1130Z, the sector 49 was a combined sector that combined sectors 45, 46, and 47 to 49 
(as indicated by the label “49+45,46,47”) while the sector 48 remained open by itself. From the time 1130Z to 
1 150Z, all five sectors were combined (as labeled “49+45,46,47,48”) before being split up at 1 150Z to three separate 
sectors (sector 45, sector 48, and a combined sector of 49+46,47). The D-side assignments is illustrated as an icon of 
a person at the start and the end of a time duration, as shown on the combined sector 49+46,47 from 1150Z to 
1230Z. The cyan line with the label “1055Z” indicates the current time. 



Figure 7. OASIS Active Plan Screen, Plan View 


The bottom graph in the Plan View showed the predicted aircraft count as a percentage of MAP for the sector 
configurations that are shown in the top graph. The graph was color-coded by the sectors and its legend is shown on 
the right of the graph. In Figure 7, the bottom graph shows the predicted aircraft count of sectors 48 and the 
combined sector 49+45,46,47 up to 1130Z, then the aircraft count for all five sectors combined, followed by the 
aircraft count in the sectors 45, 48, and combined 49+46,47. The graph also shows that the combined sector 
49+46,47 exceeded 100% MAP value around 1200Z. The user had an option to view just a select subset of the 
sectors at a time using the “Graph” button, which dimmed out the graphs for all but the selected set of sectors. 

At the very top of the screen in Figure 7, there is a label called “Loading” with a horizontal bar that is green in 
the middle. A Loading score indicated how well the traffic loads in the sectors stay within a target range. The green 
portion of the bar indicated the amount of time that the aircraft count in the sectors was in the target range. The gray 
bars on the left and right showed the amount of time that the aircraft count in the sectors was too low or too high, 
respectively. Next to the Loading score, Reconfiguration score indicated the total number of aircraft that needed to 
be handed off from one sector to another due to the sector configuration changes over the two hour duration. The 
number of impacted aircraft was also shown per configuration change, which was displayed as a number to the left 
of the UTC on the top graph. The Loading and Reconfiguration scores were designed to provide user feedback on 
how well the plan balances the traffic load and minimize the workload associated with the reconfigurations. 

The Plan View allowed the user to examine the entire sequence of sector configuration changes over the two- 
hour planning horizon, which had a large amount of information on a single screen. The NAS Monitor, in contrast, 
showed the user one sector configuration at a time (see Figure 8). The NAS Monitor View was included in OASIS 
in order to mirror the look-and-feel of the NAS Monitor view in the existing system, thereby creating a view that the 
air traffic personnel were already familiar with. 
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Figure 8. OASIS Active Plan Screen, NAS Monitor View 


The NAS Monitor View displayed the predicted aircraft count in 15 -minute increments and was color-coded: 
“green” indicated that the aircraft count was predicted to be below the MAP value for the sector, “yellow” indicated 
that the aircraft count was predicted to be above the MAP value due to pre -departure aircraft that were still on the 
ground, and “red” indicated that the aircraft count was predicted to be above the MAP because of the aircraft that 
were already in flight. The NAS Monitor View had timeline tabs on the top of the screen, with the blue dots on 
certain tabs to indicate that a sector configuration / D-side assignment changes occur at that time. The user could 
access these points where the sector configuration changes by clicking on the individual timeline tab. Each tab 
showed one particular sector configuration / D-side assignments for that time duration, the details of which were 
shown on the left side of the screen (called the Area Configuration Table). In Figure 8, the Area Configuration Table 
on the left shows that the sector configuration consists of sector 48 and a combined sector 49+45,46,47 with no D- 
sides on any of the sectors (presence of a D-side is indicated with a “D” on the right boxes as shown in Figure 9). 
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Figure 9. OASIS Active Plan Screen, Load Graph View 
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In Figure 9, a finer granularity of the traffic information is shown in the Load Graph View, which displayed the 
predicted traffic in one-minute increments. The MAP value of each sector was indicated with a red line and the 
MAP value on the right side of the graphs. Other than the one -minute granularity of the traffic information, the Load 
Graph and the NAS Monitor Views shared the same functionalities. 

These Views looked and functioned similarly for the three modes in OASIS: Active, CRP, and UGP. In the 
Active Plan Mode, the user accessed these views to review the overall plan that was being executed as well as the 
predicted traffic loads that were updated every minute against that plan. In the CRP, the user viewed the latest 
update of what the OASIS algorithm calculated to be the “best” set of plans based on the latest traffic and personnel 
information. The OASIS algorithm calculated and displayed multiple potential plan solutions. The different plan 
solutions could be accessed by pressing the “Alternate Plans” button that show up in the CRP mode (Figure 10). 
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Figure 10. OASIS CRP Mode, Alternate Plan Screen 


The Alternate Plans screen displayed up to three solutions (e.g. SI, S2, and S3), each with a timeline with dots 
that indicated sector configuration / D-side assignment changes. The numbers above the dots indicated the UTC time 
and the numbers below the dots indicated the sector configurations / D-side assignments that were altered at that 
particular time. From this Alternate Plans screen, the user could touch SI, S2, or S3 which will put the user back 
into CRP mode but displaying the newly selected algorithm solution. If the user liked the solution that was being 
viewed, s/he pressed the “Accept” button to activate that particular plan solution as the new Active Plan. 

The displays in the UGP mode functioned similarly to the CRP mode. However, in the UGP mode, the user 
could inherit configuration plans from either the Active Plan or the CRP and modify them to accomplish his/her 
needs or preference. S/he could select or drag the sectors on the display to manually combine and split the sectors. 
Similarly D-sides could be added to or subtracted from any sector that was open. The user could also interact with 
the timeline to add a new sector configuration or delete an old one, as well as moving an existing configuration from 
one time slice to another. Once the user liked the resulting plan, s/he accepted the plan to make it into the Active 
Plan. 


IV. HITL Evaluation of OASIS 

Once a functional prototype of the OASIS tool was completed, a HITL evaluation of OASIS was conducted in 
the Airspace Operations Laboratory at NASA Ames during January, 2013. The primary goals of the study were: 

1 . Collect feedback from SMEs on the acceptability of solutions using the OASIS tool 

2. Evaluate the functionality and usability of the OASIS computer-human interface 

The questions related to the OASIS algorithm-generated CRP solutions centered around 1) the acceptability of 
the CRP solutions; 2) whether the CRP solutions were “better” than the ones generated by the Supervisors; 3) 
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whether the CRP solutions worked better in some traffic scenarios but not others; and 4) which type of the multiple 
solutions, if any, were useful to the Supervisors. 

There were other questions for the UGP mode, which was mainly designed to allow the Area Supervisors to 
modify a CRP solution to result in a “better” solution when viewed from an operational perspective. The questions 
related to the UGP solutions centered around 1) whether the resulting UGP solutions were “better” than the CRP 
solutions and 2) whether the UGP solutions worked better in some traffic scenarios but not others. 

For the evaluation of the user interface, there were many interface components that were included in this initial 
prototype with an intention to eliminate some of them if the users did not find them useful. The user preferences 
were gathered from direct observation of the Area Supervisors’ usage of the OASIS tool given the various traffic 
scenarios, as well as subjective feedback gathered from user questionnaires and debrief discussions. 

V. Method 


A. Experimental Design 

The experiment was mixed factorial design. Two independent variables were used as a part of within -subjects, 
repeated measures design: OASIS tools condition, which consisted of Baseline, CRP , and UGP conditions and 
Scenarios, which consisted of Morning, Evening, Equipment , and Weather scenarios. The study was conducted 
across two weeks with four participants in each week, running slightly different versions of the OASIS algorithms. 
The two weeks / algorithm variations were analyzed as a between-groups variable. 

Following is the description of the three OASIS tools conditions: Baseline, CRP , and UGP. 

• Baseline condition - Participants were asked to examine different traffic scenarios and design their own 
solutions of sector configurations and D-side assignments for each scenario. The Baseline condition 
established the types of solutions that the Supervisors might use today without assistance from an advisory 
tool. 

• CRP condition - Participants were asked to evaluate multiple solutions that the OASIS algorithm generated 
for different traffic scenarios. They evaluated all of the algorithm solutions that the OASIS tool generated 
and provided feedback on their acceptability after each simulation run. 

• UGP condition - Participants chose one of the algorithm solutions as a starting point and then modified it 
further, if necessary, to create a “hybrid” solution that the participants thought was an improvement to the 
algorithm solutions. 

The participants examined four traffic scenarios (i.e. Morning, Evening, Weather, and Equipment Failure 
scenarios) that required sector configuration changes. The two easy scenarios (Morning and Evening) were always 
run before the difficult ones (Weather and Equipment) to allow the participants to develop some familiarity with 
OASIS before attempting the more complex traffic scenarios. The details of the traffic scenarios will be described 
further in a later section. 

There were three OASIS conditions, four traffic scenarios, and eight participants for a total of 96 data sets. The 
three OASIS conditions were run in a block design, in which the participants completed all four scenarios for each 
condition before proceeding to the next condition. Each participant worked the same sequence of scenarios. 

The experiment lasted three days per participant. The eight participants were divided into two groups of four 
participants, spanning across two weeks. The experimental conditions for the two groups were identical except for 
one variable: the nature of the algorithm solutions. At the time of the experiment, OASIS had the capability to 
generate multiple solutions for a given traffic scenario but further exploration was needed to determine what types of 
multiple solutions would be useful to the Area Supervisors. Therefore, the two groups of participants were exposed 
to two different ways that the OASIS algorithm generated the multiple solutions. The presentation of the different 
algorithm solutions across the two groups allowed us to get as much feedback as possible on the different ways that 
the advisories could be calculated. 

For Group 1 (or “Week 1”), multiple algorithm-generated solutions were presented to the participants when they 
were deemed both “good” and “significantly different” from each other (the OASIS algorithm in this condition was 
called “Path”). In this Path option used in Week 1, the OASIS algorithm calculated the “goodness” scores for all 
possible solutions, and then presented the three top solutions that were “good” (defined by a score higher than 77 out 
of a 100) and “different” (defined by solutions that had more than 30 minutes of sector configurations that did not 
match with each other). For this group, the algorithm could generate up to three solutions but could also generate 
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fewer than three if the algorithm could not find enough good solutions. The rationale for the Path option was that 
OASIS could recommend multiple solutions of sector configurations / D-side assignments that were significantly 
different from each other and yet similarly acceptable by the algorithm, and the Supervisor could choose the one that 
s/he preferred. 

For Group 2 (or “Week 2”), three algorithm solutions were generated to either 1) bias the solution to allow more 
reconfigurations to balance the traffic load, 2) bias the solution to allow fewer reconfigurations at the cost of worse 
traffic load balancing, or 3) provide a good compromise between traffic load balance and the number of 
reconfigurations. The OASIS algorithm in this condition used in Week 2 was called “Beta”. In the Beta option, three 
solutions (SI, S2, and S3) were always presented to the user. SI represented what the algorithm deemed to be a 
balanced compromise between traffic load balance and the number of reconfigurations, S2 represented the solution 
that was biased towards more reconfigurations, and S3 represented the solution with less reconfigurations. 

B. Participants 

Eight retired FAA personnel from Cleveland Center (ZOB) served as experiment participants. Their ATC 
experience spanned from 23 to 31 years with an average of 26.75 years of ATC experience. The participants had 
recently retired within 0.5 and 6 years, respectively. Week 1 participants consisted of three participants with FLM 
experience in the test area (Area 4 at ZOB). One participant had “Controller in Charge” (CIC) experience in the test 
area. Week 2 participants consisted of two participants with FLM experience in the test area, one participant with 
CIC experience in the test area and one participant with CIC experience in the Traffic Management Unit. 

C. Airspace 

The test sectors used in this study were from high altitude sectors in Area 4 of ZOB. The five test sectors (45, 
46, 47, 48 and 49) are shown in Figure 11. The sectors were stacked vertically, with ultra-high sectors 46 and 45 
located at FL360 and above, super-high sectors 47 and 49 from FL310 to FL350, and a high altitude sector 48 from 
FL240 to FL300. The flows in the test scenarios consisted mostly of en route East- West flow going in and out of 
New York ARTCC (ZNY) and Chicago ARTCC (ZAU) with some arrivals and departures to and from the local 
area airports such as Detroit and Cleveland. 



Figure 11. Test Airspace: Area 4 in ZOB 


D. Traffic Scenarios 
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There were four traffic scenarios based on ZOB Area 4 sectors. The scenarios were developed from the use 
cases that were created earlier in the project. The simulated traffic was based on actual traffic data from ZOB 
operations and modified further in order to create trigger events that could test the OASIS functionalities. 

The four scenarios were called Morning, Evening, Equipment Failure, and Weather scenarios. Morning and 
Evening scenarios were relatively easy scenarios that were expected to occur daily at the beginning and near the end 
of the mid-shift (2230 to 0630 local time). The Morning scenario was the easiest scenario. As the mid-shift crew 
wound down on their shift and the morning crew of controllers started to come in, the traffic ramped up gradually 
and matched the personnel available. The Evening scenario focused on the end of the swing shift as the traffic 
ramped down and the swing shift crew started to go home. At about 0200Z (2200 local), the swing shift supervisor 
receives an alert from OASIS that a new recommended plan is available. The reconfigurations coincide with the 
reduction of controller resources due to their shift end times. 

Weather and Equipment scenarios were designed to create off-nominal situations that both OASIS and the Area 
Supervisors would need to solve by changing the sector configurations. The Equipment scenario assumed that the 
Operational Manager in Charge has contacted Area 4 and requested release of Sector 47 for required maintenance. 
The power supplies at both the D-side and the R-side must be replaced. Maintenance must be performed to prevent 
a catastrophic failure. The Supervisor, who is being relieved, has approved the release of Sector 47 from 203 0Z to 
2130Z. OASIS will provide a new recommended plan based on the loss of Sector 47 for that period of time. 

The Weather scenario started at the same time as the Morning scenario, in which the morning-shift controllers 
were coming on duty as the traffic ramped up. However, a weather cell to the north of Area 4 forced extra traffic 
into the test airspace (Figure 12). Area Supervisor has limited options in terms of sector configurations due to lack 
of controllers on duty in the next 30 minutes to an hour but must do the best s/he can. Following is a more detailed 
back-story given to the participants: 

The first morning rush from the New York Metroplex is underway and a steady flow of aircraft is flying 
toward ZOB Area 4 from New York Center. However, as the morning progresses, a line of thunderstorms 
has developed from Lake Michigan to Lake Huron and now pilots are consistently asking for deviations 
south of track to avoid weather. Traffic Management has determined that a SWAP route for aircraft leaving 
Boston, Montreal, Toronto, and vicinity is required. As the new route is implemented, the NAS Monitor 
Panel has begun to show a significant increase in Area 4 traffic and OASIS provides a new recommended 
plan. 



Figure 12. Weather Scenario 


E. Software and Equipment 

The workstations for the four participants were spread across two experimental rooms with two stations in each 
room. Each participant’s test area consisted of two types of equipment. The first equipment was a portable tablet 
computer with a touch screen that hosted the OASIS software (see Figure 6). For the tablet, an ASUS Transformer 
Pad TF700T model running Android 4.0 was chosen for its fast processor speed (1.6 GHz NVIDIA Tegra 3 Quad- 
Core processor) and high graphics resolution (1920x1200). The OASIS software was written in Java. 

The second set of equipment provided the air traffic information to the participants. The information was 
presented across two 28 inch Barco displays located side-by-side with PC keyboard and mouse as input devices. The 
Barco displays are 20 inch by 20 inch, high-resolution displays that are used by the air traffic controllers in today’s 
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operations. The traffic information shown on the Barco displays were provided by a NASA-developed software 
package called Multi-Aircraft Control System (MACS). 8 This is a Java based software package developed at the 
NASA Ames Research Center. MACS can emulate many of today’s and NextGen air traffic management 
capabilities. For this study, many of the traffic related tools such as a real-time display of traffic and traffic load 
graphs/table were utilized to provide the traffic information to the participants. 

There were two MACS displays at each of the four positions. The left side display showed (see Figure 13): 

• Baseline Load Table Window (Top Left): Traffic load table for each of the individual 5 sectors in Area 4, 
regardless of whether the sectors are combined or not. This window mimicked the “baseline” function in the 
current Traffic Flow Management System that exist in the field. 

• Display System Replacement (DSR) 4 (Top Center): Simulated Traffic Situation Display that shows 
different traffic flows into the DC area 

• DSR3 (Top Right): Simulated TSD that shows different traffic flows between the Chicago and New York 
areas 

• DSR2 (Middle): Simulated TSD3 that shows all traffic in Area 4, segregated by altitudes (FL240-300, 

FL3 10-350, FL360-600) 

• Time in Sector (TIS) and List IFR Flight Plan (LIFP) data (Bottom): TIS and LIFP data for each sector for 
both initial and baseline configurations 
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The right side display showed (see Figure 14): 

• Multi Load Graph Window (Top Left): traffic load graphs for each of the sectors that are opened at the start 
of the simulation run 

• Load Table Window (Top Right): traffic load table for each of the open sectors 

• DSR1 (Bottom): DSR that interacts with the participant actions via Load Graphs, Load Table, aircraft 
Filters, and the Computer Readout Device 



Figure 14. Right Side MACS Display 


F. Procedures 

For each condition, four simulation runs consisting of four different traffic scenarios in the ZOB Area 4 were 
presented to the participants. The four traffic scenarios / runs were blocked according to experimental condition such 
that the Baseline condition block was presented first, followed by the CRP condition, and finally the UGP condition 
block. This particular order and the block design were used so that the participants’ Baseline solutions without 
OASIS would not be influenced by previewing the OASIS algorithm solutions. The same four traffic scenarios were 
used across the different conditions. 

Each participant was accompanied by an observer who collected data and conducted individual interviews as 
needed. In each simulation room, there were two wall charts. One was a three-dimensional sector plan for the ZOB 
area and the second chart was a top-down view of the area. There was a “relief briefing” prior to the start of each 
experimental condition during which the participant was provided with printouts showing staffing (area schedule 
worksheet modeled after WebScheduler), a briefing sheet, and weather information. 
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1 . Baseline Condition 

When the participant sat down at the MACS workstations for the Baseline condition, the screens showed an 
initial configuration (a different one for each scenario). These were static “screen shots” showing what the traffic 
pattern would look like after about 90 seconds from startup. These screen shots were visible during the initial 
briefing of the traffic problem. When the session was about to begin, the static screens were hidden, and the actual 
MACS displays were shown instead (initially with no traffic). When the participant was ready, MACS was started 
on both screens at the same time. A traffic sample was run for each of the four scenarios so that the participant 
could interact with MACS to look for information. We recorded the MACS screen and participant audio using 
Camtasia. Camtasia was started at the beginning of each run. 

During the Baseline condition, the participant was asked to construct his/her own two-hour plan of sector 
configuration changes and D-side assignments. OASIS was not used. The participant was given a paper form to 
record the plan. After the participants completed each run, an online questionnaire was displayed on the right side 
MACS screen. After completing this, they took a break and the researchers shut down the MACS stations and the 
Camtasia recording. Camtasia recordings were saved manually to a predefined folder location with a specific 
naming convention to identify each position and run. MACS was then configured for the next run. 

2. CRP Condition 

In the CRP condition, the initial briefing and the start up procedure was identical to the Baseline condition. 
Once the briefing was over and the participant was ready, MACS was started on both screens at the same time. 
OASIS was also initiated. The times on all three screens were calibrated to be the same. A traffic sample was run 
for each of the four scenarios and the participant interacted with MACS to search for information. We recorded this 
audio and the MACS screen using Camtasia. Camtasia recording was also started at the beginning of each run. 

The OASIS tablets had a setup screen where the researchers set the condition and scenario. The opening screen 
was set to the Plan View in the Active Plan mode. On the OASIS screen, the participant was asked to press the CRP 
button in order to take him/her to the OASIS advisory solutions. S/he stepped through all three Views and all 
configuration changes for all of the available solutions. The participant selected and evaluated up to three solutions 
and took notes during the evaluation. Once the evaluation was completed, s/he was asked to choose the “best” 
solution and make it the Active Plan. Once this action was completed, the simulation run was over. 

After the participants completed each run, an online questionnaire was displayed on the right side MACS 
screen. In addition to the questions asked in the Baseline condition, the participant was asked to evaluate the 
acceptability of the overall CRP, and each of its included configurations. After completing the questionnaire, they 
took a break and the researchers shut down the MACS stations, OASIS tablets, and the Camtasia recording. 
Camtasia recordings were saved manually at a predefined folder location with a specific naming convention to 
identify each position and the run. MACS and OASIS were then reconfigured for the next run. 

3. UGP Condition 

The initial briefing and the startup procedure was identical to the CRP condition. Once the briefing was over 
and the participant was ready, MACS and OASIS were initiated and calibrated with each other. Camtasia recording 
was also started to capture the MACS screen and the audio. 

Similar to the CRP condition, the participant was asked to press the CRP button and quickly evaluate the 
OASIS advisory solutions in order to pick the one that they wanted to use as the basis for the user modification of 
the plan. The participant was then asked to push the button to go to the UGP mode. S/he then modified the CRP 
plan, using the UGP tools. Once the participant finished modifying the plan, s/he was asked to “accept” the UGP 
solution and make it the Active Plan. Once this action was completed, the simulation run was over. At that point, 
OASIS automatically recorded the final UGP to a time-stamped file on the ASUS tablet at the end of the 
participants’ interactions with the tool. After the participants completed each run, they completed an online 
questionnaire in the same way as in the first two conditions. 

4. Usability Assessment 

After the three conditions and four scenarios were completed, participants assessed the usability of OASIS 
using an online survey form with rating scales or questions for each function. The evaluation was completed with 
the participant group together in a conference room. Each participant used their OASIS tablet to view each of the 
screens and filled out the usability questionnaires on a laptop. The OASIS screens were also projected in the room 
for easy reference during this part of the evaluation. 
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Due to similarities between the different Views across the different Modes, the participants were asked to 
evaluate only a subset of the Views across the Modes. The OASIS screen that were evaluated were: 1) CRP: Plan 
View, NAS Monitor View, Load Graph View; 2) UGP: NAS Monitor View and Load Graph View; and 3) 
Constraints screen. The Constraints screen was a prototype that allowed the user to modify the available personnel 
and workstations that the OASIS algorithm uses as its constraints. Although the participants did not directly interact 
with the interface, their feedback was elicited to refine the interface and the functionality of this screen. 

5. OASIS Functionality Evaluation 

After the usability session, participants were asked for their feedback on the effectiveness of the various screens 
and capabilities. The laptops were used again to administer an online survey for the functionality questionnaire. 

6. Debrief Discussion 

The final activity in the study was to have a group discussion on the overall concept, the user interface, and the 
underlying algorithm-generated plan solutions. The discussion covered many of the topics that were already elicited 
in the questionnaires but the participants could elaborate on the details of their comments in this setting. 

VI. Results 

One of the main goals of the study was to collect feedback from the participants on the acceptability of 
solutions generated using the OASIS tool. To answer this question, the participants were asked to rate the 
acceptability of the configuration solutions after each simulation run. In the Baseline condition, they were rating the 
acceptability of their own solutions. In the CRP condition, they rated all of the algorithm generated advisory 
solutions. In the UGP condition, they rated the final solution that resulted from the user modification of the OASIS 
advisory solutions. Other questions related to the “goodness” of the CRP and UGP solutions were also posed to the 
participants to elicit feedback. The subjective feedback was supplemented by objective data that were collected from 
the OASIS tool. Key results are described in the following sections. 

A. Baseline Condition 

In the Baseline condition, the participants used MACS data and other information to generate their sector 
configurations and D-side assignments for subsequent two hours. Figure 15 shows the acceptability of the resulting 
plans. Rating scale was 1 to 5, with 5 indicating high acceptability and 1 indicating low acceptability. 
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Figure 15. Acceptability of the Participants’ Plans without OASIS in Baseline condition 

(error bar = +/- one standard error) 

Overall, the results showed that the acceptability of the manually-generated plans was high across all scenarios. 
In general, the acceptability of the plans were higher for the “easy” traffic scenarios (e.g. Morning) and lower for the 
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difficult scenarios (e.g. Weather), but repeated measures within-subjects Analysis of Variance (ANOVA) resulted in 
no significant differences. F(3,18) = 1.84, p = .18. For the Baseline condition in which the participants constructed 
their own plans without OASIS, the two groups in Week 1 and 2 performed the exact same tasks and therefore 
should have no difference in their results. Although the results suggest some differences between Week 1 and 2 
participants, the between-groups analysis showed no significant differences. F(l,6) = .22, p = .65. 

During the post-run questionnaire survey, if a participant gave a low acceptability rating for a given simulation 
run, s/he was asked to explain. In the Baseline condition, there was only one participant in Week 1 who gave a low 
rating for his own plan solution for the Weather scenario. He explained that the low rating resulted from the lack of 
controller availability during high traffic volume did not allow him to construct an adequately acceptable plan. 

Figure 16 provides feedback on the sources of information that were considered most helpful in creating the 
plans when the participants constructed their own plans without OASIS. The information is ranked from the highest 
to lowest percentages of “YES” responses across participants. The participants responded that they had sufficient 
information from the handouts and DSR screen to generate their plans. When they were asked to provide any 
missing information, two participants wanted to easily distinguish the arrivals and departures to local airports in 
Area 4 (i.e. Detroit and Cleveland) so that they can account for the traffic complexity of transitioning aircraft. 


Did you have sufficient information? 

Baseline Load Table 
Area schedule worksheet 
Load Table 
Area briefing sheet 
DSR2 Aircraft by Altitude stratum 
DSR1 Basic View 
Load Graph 
Time in Sector Chart 
DSR3 Aircraft by Airport - NY to/from Chicago 
DSR4 Aircraft by Airport - DC and Philadelphia 

LIFP Table 

0 10 20 30 40 50 60 70 80 90 100 

Percent of "YES" Responses 

Figure 16. Information used by the participants to construct their own plans in Baseline condition 

The most useful data appeared to be the Baseline Load Table (load table of the five non-combined sectors in the 
Area), Load Table (load table of the open sectors), Area schedule worksheet (controller time schedule to determine 
number of available controllers), and the Area briefing sheet (briefing material of the traffic and personnel situation 
at the shift change). There were some minor discrepancies from the Week 1 to Week 2 participants but the overall 
ranking of the information was consistent between the two groups, with an exception of TIS Chart was used by more 
than 50% of Week 1 participants but none from Week 2 participants. 

B. CRP Condition 

One of the important questions for the evaluation of OASIS was the acceptability of the CRPs. A key question 
was to know whether the participants found the algorithm generated OASIS advisories to be workable and made 
sense to them in the operational context. In the CRP condition, the participants were asked to evaluate up to three 
OASIS advisory solutions and pick the best solution at the end of each simulation run. Figures 17 shows the 
participants’ acceptability ratings for the best CRP solution across four traffic scenarios. Similar to the Baseline 
condition, the overall results show that the acceptability of the best CRPs were high across all scenarios. Repeated 
measures within-subjects ANOVA analysis showed marginal significance of Scenarios. F(3,18) = 2.44, p = .098. No 
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other main or interaction effects were significant. In general, the acceptability of the plans was higher for the “easy” 
traffic scenarios (e.g. Morning) and lower for the difficult scenarios (e.g. Weather). Simple within -subjects contrasts 
resulted in significantly higher acceptability ratings for the Morning scenario compared to the Weather scenario. 
F(l,6) = 13.7, p = .010. Between the two weeks, the participants in each week assessed different algorithm methods 
for generating the multiple CRP solutions (i.e. Path vs. Beta). In spite of the different methods, the between-groups 
analysis showed no significant differences between the two weeks. F(l,6) = .63, p = .45. 
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■ Week 1 / Path 
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Figure 17. Acceptability of the best CRP in each scenario 
(Note: Morning and Equipment conditions for Week 1 / Path had only one CRP) 


Figure 15 and 17 illustrate that both manually-generated solutions without OASIS and the algorithm generated 
CRP solutions were highly acceptable. A direct comparison between these two ratings showed that the average 
acceptability rating in the CRP condition (M = 4.4, SD = .76) was marginally higher than that of the Baseline 
condition (M = 4.1, SD = .79). F(l,6) = 4.8, p = .072. Similar to prior results, the Scenarios significantly differed 
from each other (F(3,18) = 4.5, p = .016) and there were no significant difference across the two weeks. Figure 18 
shows the average acceptability ratings across Scenarios and Conditions. 



Morning Evening Equipment 

Scenario 


Weather 


■ Baseline 

■ CRP 


Figure 18. Comparison of acceptability ratings between Baseline and the best CRP solutions across scenarios 


The acceptability ratings for the plan solutions in Baseline and CRP conditions were both high and similar to 
each other, which leads to a question whether the OASIS algorithm is generating solutions that mimic the manually 
generated solutions without OASIS. To answer this question, the sector configurations between Baseline and CRP 
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condition per each simulation run were matched and compared for their similarities. The “similarity” between the 
two matched configuration plans was operationalized by simply counting the time duration in which the sector 
configurations between the two conditions provided an exact match. In this analysis, mismatches in D-side 
assignments were ignored. Figure 19 shows the results of this analysis. The results show that except in the 
Equipment scenario, only about 50% of the sector configurations matched between Baseline and CRP, suggesting 
that the OASIS provided acceptable but different solutions to the users. The equipment scenario was an exception, 
perhaps because the loss of a workstation / sector constrained the problem such that only a limited set of viable 
sector configuration options were available. 



Figure 19. Sector Configuration Similarities between Baseline plans and CRPs 

Further analysis is needed to understand the nature of the difference between the two conditions but the total 
number of reconfigurations performed by the participants vs. OASIS algorithm provides some insight. Figure 20 
shows the total number of reconfigurations that were done during the study in the two conditions. For both Path and 
Beta options of generating algorithm solutions, the OASIS advisory solutions in the CRP condition resulted in fewer 
number of reconfigurations compared to the manually generated ones in the Baseline condition. 



Figure 20. Number of reconfigurations 
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The best CRP solutions were rated higher than the Baseline solutions, suggesting that the OASIS algorithm 
generated solutions that were as good as or better than the ones that the participants generated themselves. However, 
this was only true for the best CRP solutions. When OASIS generated multiple CRP solutions, the ones that the 
participants did not choose often had lower ratings. Figure 21 shows the ratings for the three CRP solutions 
compared to the best CRP solutions, combined across Scenarios. Repeated measures within-subjects ANOVA 
analysis showed significant differences in ratings for the solutions. F(3,18) = 6.31, p = .004. The interaction effects 
were not significant. Simple within-subjects contrasts resulted in significantly higher acceptability ratings for the 
best solution compared S2 (F(l,6) = 1 1.7, p = .014) and S3 (F(l,6) = 30.3, p = .002) but not SI (F(l,6) = 2.82, p = 
.144). Although different algorithm methods were used to generate the multiple CRP solutions (i.e. Path vs. Beta) 
between the two weeks, the between-groups analysis showed no significant differences between the two weeks. 
F(l,6) = .61, p = .46. 
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Alternative Solutions in CRP Mode 


Figure 21. Acceptability of the three alternative solutions the best solution chosen in CRP mode 
(Note: Morning and Equipment conditions for Week 1 / Path had only one CRP) 


When OASIS generates multiple solutions, S 1 was always designated as what the algorithm has determined to 
be the “best” solution. For the Path option, S2 and S3 were the next set of alternative solutions based on the 
mathematical calculations that the algorithm generates. For the Beta option, SI provided a solution with a good 
compromise of balancing the traffic load across sectors without “too much” reconfigurations, S2 with a solution 
with more reconfigurations but better traffic load balance, and S3 with a solution with less reconfiguration. For both 
Path and Beta options, SI resulted in higher acceptability ratings than S2 and S3, suggesting that SI provided the 
best solution in most cases for both Path and Beta options (Figure 21). 

However, the results also indicate that there is value in providing multiple CRP solutions instead of providing 
just SI. The acceptability of the best solution was higher than SI, S2, and S3, suggesting that the participants did not 
always pick SI as their best solution. Also when the participants were asked to pick the best solution for each 
scenario during the post-run questionnaire, each participant picked different solutions across scenarios. Table 1 
below shows the percentages of times that the participants picked each solution as the best solution. Morning and 
Equipment scenarios under Path option only had one solution and therefore were excluded from the results. The 
results show that when three solutions were available, the solutions were chosen fairly evenly across the scenarios 
and participants for both and Path and Beta options. 


Table 1. Percentage of SI, S2, and S3 being selected by the participants 
(Note: Morning and Evening scenarios in Path option only had SI and therefore were excluded from the analysis.) 



SI 

S2 

S3 

Path 

37.5 

37.5 

25 

Beta 

37.5 

18.75 

43.75 
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Although the data suggest that providing multiple solutions may be valuable to the users, comments from the 
participants suggested that it might be too difficult to process and compare multiple plans. They commented that 
each plan had many data sources to consider during evaluation. The scenarios had a lot of traffic, and the CRP had 
many configuration changes. In addition, the CRP solutions sometimes updated over time as the traffic information 
changes, making it difficult to keep track of the different plans. In Beta condition, some plans could also be quite 
similar to others, also making it difficult to keep track of different plans. 

When the participants gave a low acceptability rating (i.e. 1 or 2 rating) for the CRP solutions, they were asked 
to provide the reason for the low ratings. Table 2 summarizes the percentage of times that SI, S2, and S3 solutions 
were rated low across the scenarios. In general, the Morning scenario had 100% acceptable solutions, the Weather 
scenario had the highest instances of low acceptable solutions, and S3 solution for the Evening scenario was not 
very acceptable. 


Table 2. Percentage of low acceptability ratings for SI, S2, and S3 
(Note: Morning and Evening scenarios in Path option only had SI.) 



SI 

S2 

S3 

Path 

Morning 

0 

NA 

NA 


Evening 

0 

25 

50 


Equipment 

0 

NA 

NA 


Weather 

25 

50 

25 

Beta 

Morning 

0 

0 

0 


Evening 

25 

0 

100 


Equipment 

25 

25 

0 


Weather 

50 

50 

50 


When the participants were asked to provide the reasons for their low ratings, a common set of reasons emerged 
across the different traffic scenarios. The main reasons were 1) the solution could not prevent the traffic to exceed 
the MAP value; 2) D-side should have been added with high traffic levels; and 3) too many reconfiguration changes 
in a short amount of time. The inadequate solution that did not prevent the traffic from exceeding MAP or did not 
add D-side was a common theme for the Weather scenario which had a shortage of controllers for the traffic level at 
the beginning of the scenario, making it difficult to find an adequate solution during that time period. 

Additional questions were asked about the SI plan, which was hypothesized to be the best of the three plans at 
the start of the study, in order to gain further insight into the acceptability of the algorithm generated OASIS 
advisory solutions. Figure 22 presents these results. Overall, the participants indicated that OASIS provided 
acceptable solution in terms of airspace reconfiguration, D-side changes, and the traffic workload. There were no 
statistically significant differences between the Path and Beta conditions for any of the measures. 



Figure 22. Evaluations of the CRP Path and Beta conditions, collapsed across scenarios 


Finally, Figure 23 shows the information sources the participants used to evaluate the CRPs. As in the Baseline 
Condition, they reported having sufficient information. The Area briefing sheet was counted as useful, but there 
was not much reference to other sources of information in the handout or DSR displays. This may not be too 
surprising because the same traffic scenarios were used in Baseline and CRP conditions and therefore the 
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participants were already familiar with the traffic. A second reason may be that OASIS also provided traffic related 
information, thereby making the other information sources redundant. 


Did you have sufficient information? 

Area briefing sheet 
Load Table 
Area schedule worksheet 
Baseline Load Table 
Load Graph 
LIFP Table 
Time in Sector Chart 
DSR4 Aircraft by Airport - DC and Philadelphia 
DSR3 Aircraft by Airport - NY to/from Chicago 
DSR2 Aircraft by Altitude stratum 
DSR1 Basic View 

0 10 20 30 40 50 60 70 80 90 100 

Percent of "YES" Responses 



■ Week 1 / Path 

■ Week 2 / Beta 

■ Both Weeks Avg. 


Figure 23. Information sources used when evaluating the CRPs 


C. UGP Condition 

The evaluation of the CRP mode was focused on whether the OASIS algorithm could generate solutions that 
were “better” than the solutions without OASIS. In the UGP mode, the evaluation focused on whether the 
participants could modify one of the CRP solutions to make UGP solutions that would increase their acceptability to 
the Area Supervisors. Figure 24 shows that the acceptability ratings of the UGP solutions were high for the final 
UGP solutions, similar to those in the Baseline and CRP conditions. There were no statistically significant 
differences for acceptability between conditions or scenarios. There was only one instance in the Equipment 
scenario in Week 2 that resulted in a low rating in the UGP condition. The reason for the low ratings was that the 
participant tried but failed to construct a plan that could keep one of the sectors from exceeding its MAP value for a 
10-minute duration. 
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Figure 24. Acceptability of final UGPs for each scenario 
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Figure 25 shows a direct comparison between CRP and UGP acceptability ratings to evaluate if UGP mode 
could improve the solutions from the CRP mode. The results showed no significant difference in the ratings between 
the two conditions across the four scenarios. The average acceptability rating in the CRP condition (M = 4.4, SD = 
.76) was higher than that of the UGP condition (M = 4.3, SD = .85) but the difference was not significant. The 
results suggest that contrary to the hypothesis, the UGP mode did not improve the acceptability of the CRP 
solutions. 



Scenario 


■ CRP 

■ UGP 


Figure 25. Comparison of acceptability ratings between the best CRP and the final UGP solutions across 

scenarios 


The acceptability ratings for the plan solutions in CRP and UGP conditions were quite similar to each other. 
Further analysis was done to see if the similarity in ratings were due to the actual similarities in the sector 
configurations between the two conditions. Figure 26 shows the percent of exact matches of sector configurations 
between the UGP solutions and the initial CRPs from which the UGPs were constructed. The results show that the 
sector configurations matched about 80% - 90%, indicating that the participants made only minor adjustments to the 
original CRP solutions. 



Figure 26. Sector Configuration Similarities between UGPs and the initial CRPs 
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Although the UGP solutions did not improve the acceptability, the participants nevertheless reported that it was 
helpful to have the ability to construct UGP solution by modifying the CRP solutions (see Figure 27). They reported 
that only a moderate amount of changes were required to modify the CRP to become the final, accepted UGP. In 
addition, they reported a moderate level of approval for the alternative plans being helpful and unique, 


Helpful to Modify CRP to Make UGP? 
Many Changes to CRP to Make UGP? 
Were Alternative Plans (S2 and S3) Helpful? 
Alternative Plans Different Enough from SI? 



Rating 


■ Week 1 / Path 

■ Week 2 / Beta 


Figure 27. Rating of the UGPs on several measures related to UGP 


In the UGP condition, the participants were able to consult MACS and also the plan information presented in 
OASIS. Figure 28 shows the information sources the participants used to create the UGPs. As with the CRP results, 
participants said they had sufficient information, and relied on the Area briefing sheet, Baseline Load Table, and 
Area schedule worksheet. Other data sources (apart from OASIS) were not used much. 


Did you have sufficient information? 

Area briefing sheet 
Baseline Load Table 
Area schedule worksheet 
Load Table 
Load Graph 
Time in Sector Chart 
DSR1 Basic View 
LIFP Table 

DSR4 Aircraft by Airport - DC and Philadelphia 
DSR3 Aircraft by Airport - NY to/from Chicago 
DSR2 Aircraft by Altitude stratum 



i Week 1 / Path 
l Week 2 / Beta 
Both Weeks Avg. 


0 10 20 30 40 50 60 70 80 90 100 

Percent of "YES" Responses 


Figure 28. Information sources used when evaluating the UGPs 
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D. OASIS Usability 

Given that this HITL evaluation was the first for the OASIS tool prototype, the feedback on the usability of the 
tool was a crucial component of the study. There were three sources of data for OASIS usability. The participants 
provided ratings on OASIS screens and views. We also asked questions in the post-run surveys and the participants 
wrote comments and verbally responded to questions in the debriefing sessions. 

Prior to the study, the research team debated on the relative value of the three main OASIS screens (or 
“views”): Plan View, NAS Monitor View and Load Graph View. In order to find out which of the three 
screens/views were helpful, participants were asked which of the three views were helpful to choose or construct a 
CRP or UGP solution, respectively, after each simulation run. Figure 29 shows the percent of “YES” responses to 
the question for the three OASIS views. The results suggest that the participants actually used all three views 
extensively. The usage of the Plan View seems similar for CRP and UGP condition but NAS Monitor View was 
used more often in the UGP compared to the CRP mode, and the pattern was reversed for the Load Graph View. The 
results seem to be consistent with the observation during the study that the modification of sector configuration 
plans in the UGP mode was conducted primarily in the NAS Monitor View. 
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Figure 29. Ratings of helpfulness of OASIS views 


During the post-simulation usability survey, the participants were asked their feedback on the individual 
components of the OASIS interface, such as the Timeline, NAS Monitor Table, and Area Configuration Table. The 
summary of the survey results are in Appendix B. The ratings results showed that participants gave high ratings for 
all of the major components of the OASIS tool (i.e. between 4 and 5 on a scale of 1 to 5). 

Appendix B also has a summary of participant comments included in the written questionnaires and debriefing 
discussions. These comments provide additional feedback on how to improve the tool. The feedback have been 
collected and prioritized for the next iteration of the interface design. Following are some of the main feedback on 
the user interface: 

• The Plan View, which displayed the entire two-hour sector configuration plans and the associated traffic 
load, was used often by some participants and not at all by others. Because this view had so much 
information, some participants found it hard to decipher. In particular, the traffic information at the bottom 
of the screen had too much data and took a while to digest. On the other hand, other participants wanted to 
put even more functionality in this view. One consensus from the participants was that they would like to 
see the staffing information on the screen, such as the number of controllers used per configuration and the 
number of controllers available. 

• The NAS Monitor and Load Graph Views had similar layouts: Timeline at the top, Area Configuration 
Table on the left, which shows the sector configuration and D-side assignments at a particular time segment, 
and the predicted traffic data on the right (NAS Monitor View has a table with 15 -minute increments and 
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Load Graph View has graphs with 1 -minute increments). For Timeline on both Views, participants wanted 
the selection of a particular time segment to be easier than the current implementation, which has very small 
tabs due to lack of space on the screen. 

• Comparing the NAS Monitor vs. Load Graph View, some preferred to use the NAS Monitor View while 
others preferred the Load Graph View. They liked the NAS Monitor View since it was modeled after a tool 
in today’s operations and therefore easy to understand. Some of the participants also found Load Graph 
View to be useful in displaying traffic “peaks and valleys” over time. 

• Participants also gave useful feedback on improvements on the tool’s ability to manually move the section 
configuration, D-side assignments, and the time segments in the UGP mode. They suggested improving the 
ability to move a section configuration from one time segment to another, as well as adding different ways 
to change D-side assignments. One participant also expressed the desire to be able to make the UGP 
modifications in the Plan View instead of in NAS Monitor or Load Graph Views which is the current design 
of the tool 

E. OASIS Functionality 

1 . OASIS Advisories 

During the post-simulation survey, the participants were asked additional questions with regard to the OASIS 
tool functionalities. Follow-up questions about OASIS CRP solutions provided further insights into the users’ 
perspectives of the advisory solutions. From the eight participants in the study, seven participants reported that the 
advisories proposed airspace configurations that they had not considered and that those configurations were useful. 
Seven also reported that the OASIS advisories helped them to plan ahead compared to the operations without 
OASIS. Although they reported that the advisories were helpful, they also listed the factors that OASIS does not 
consider in making the advisories that the Area Supervisors must consider in making their decisions. These factors 
were: 

• Controller skill level/qualifications 

• Access to staff during breaks 

• Turbulence in a sector may cause too many aircraft in one sector if pilots change altitude 

• Need to complete position relief briefings 15 minutes before traffic increases 

• Does not take into account high workload due to, for example, vectoring 

• Include the tracker position 

• Multiple sector reconfigurations in a short period of time 

• Sector complexity 

• Position overlap (e.g., should not remove a D-side until after sectors are combined or split) 

• Weather 

When they were asked how many advisory plans that they would like to see, the participants in the Path option 
reported an average of 2.8 plans (SD = .5) and those in the Beta option reported an average of 1.8 plans (SD = .5). 
They thought that the response time for generating new OASIS plan was highly acceptable (rating M = 4.6 out of 5, 
SD = .52). Finally, the current implementation of OASIS did not allow a sector configuration change during the first 
15 minutes of the plan with an assumption that the Area Supervisors would be working at least 15 minutes into the 
future when s/he is planning to make changes. When the participants were asked if the 15 minute limit was 
necessary, six of the eight participants did not think so. 

2. Loading and Reconfiguration Scores 

When OASIS provided CRP solutions, the tool provided two types of scores to provide feedback to the users on 
the two main factors that the algorithm is managing: balancing traffic load and lowering the workload cost of sector 
reconfiguration. Loading score was presented to the users to illustrate how well the traffic loads in the sectors are 
within a target range of being not too high or low. All eight participants reported that they understood the meaning 
of the Loading score in OASIS but they found it only moderately helpful in making sector configuration decisions 
(rating M = 2.8 out of 5, SD = 1.5). For the participants who found the Loading score to be helpful, they reported 
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that the score made it easier to visualize the factor and was helpful to identify when the traffic load was close to the 
high end of the traffic threshold. However, many did not find it that useful. 

Reconfiguration score was presented to the users to illustrate how many aircraft will need to be handed off from 
one sector to another during a sector configuration change. This score was used as a proxy for the reconfiguration 
cost that the OASIS algorithm was evaluating internally. All eight participants reported that they understood the 
meaning of the Reconfiguration score in OASIS and they gave moderate to high ratings in its helpfulness in making 
sector configuration decisions (rating M = 3.8 out of 5, SD = 1.1). Some participants reported that it helped to 
identify a time segment when the traffic was low to make the sector configuration change and the score made it 
easier to visualize the factor. However, others reported that it did not factor into their decision making one of the 
participants indicated that he would not base his decision on it because it is not that difficult to reconfigure airspace 
and move people around. 


VII. Discussion 

The OASIS tool provided two main functionalities. The first functionality was for the OASIS algorithm to take 
the predicted traffic data and the available personnel and create a set of sector configurations and D-side 
assignments over a two-hour time horizon. The main questions related to the algorithm generated OASIS advisories 
were if they were acceptable and provided solutions that were potentially better than the Supervisors could generate 
on their own without OASIS. 

The overall results indicated that the OASIS advisory solutions performed very well. Although OASIS did not 
consider many of the factors that the Area Supervisors take into account in making their decisions, the participants 
nevertheless found the OASIS advisories very helpful and the number of airspace reconfiguration and D-side 
assignments to be acceptable. OASIS was able to provide solutions that the participants did not consider before and 
it also allowed them to plan further ahead than normal. In addition, a comparison between the Baseline and CRP 
condition showed that although both conditions generated highly acceptable solutions, the best CRP solutions were 
marginally more acceptable than the Baseline solutions. 

Some of the additional factors for decision making that participants listed could theoretically be added in a 
future iteration of OASIS. For example, the participants suggested removing the D -sides after sectors are combined 
or split, which could be added as a constraint in OASIS. Sector complexity could be considered by replacing the 
predicted aircraft count with a combination of traffic complexity metrics. Controller skill level and more detailed 
access to staff schedule could also be added to OASIS, although there are practical limitations on getting access to 
those databases. 

In the study, the participants were given multiple CRP solutions to choose from. Between Week 1 and 2, two 
groups of participants were given different methods (i.e. Path and Beta) of generating the multiple plan solutions. 
When they were asked how many plans that they would like to see, the group under Path option suggested having 
around three plans as currently designed while the group under Beta option suggested having around two plans. 
Their acceptability ratings data support the benefit of having multiple plan solutions. When they were presented with 
SI, S2, and S3 plans with SI designated by the tool as the “best” plan, they rated on the average that SI was the 
better than S2 or S3. However, some participants also picked S2 or S3 as the best solution in some of the traffic 
scenarios, confirming that having multiple solutions gives the user flexibility to pick different solutions depending 
on the context. The main caution against having too many solutions is that the amount of information that the Area 
Supervisor would need to digest for each plan solution is quite large and therefore too many solutions could 
overload the user. In the next iteration, limiting the multiple solutions to two instead of three is something that 
should be explored. 

In addition to the algorithm generated advisories, OASIS also gave the user the flexibility to modify the existing 
solutions to make their own UGP solutions. The rationale was that since OASIS does not take all of the factors that 
are considered for sector configuration changes and D-side assignments into account, the user can tweak the 
algorithm generated solutions to make any necessary adjustments. The results from the study on the UGP mode 
were mixed. Although the participants reported that it was helpful to modify the existing plan solutions to make new 
ones, they could not significantly improve upon the algorithm-generated solutions within the study. The results 
suggest that although the users liked the flexibility of having the UGP mode, OASIS algorithm presented good CRP 
solutions that were acceptable as-is and did not need further modifications. 

This HITL study provided the first formal evaluation of the initial prototype of the OASIS tool. Many of the 
interface components were added to elicit feedback on their usability and usefulness to guide the future design. For 
example, three screens/views were presented to the participants to see which of them were useful. They reported that 
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all three views were useful even though some thought that the Plan View had too much information and was 
difficult to decipher. Many liked the NAS Monitor View and it was used often, especially when they were 
constructing their own plans in the UGP mode. Finally, they rated most of the interface components highly and 
thought that it was easy to use and understand. They also provided valuable feedback on the components that could 
work better and suggestions on how to improve them. Overall, the feedback on the OASIS interface was very 
positive. 


VIII. Conclusion 

Based on the promise of past research in DAC that showed potential benefits of an advisory tool that can 
suggest sector configurations and D-side assignments based on the traffic and personnel data, a prototype tool called 
OASIS was born. A team of researchers spent the past year going through a design process that outlined the 
information requirements, created use cases, and iterated on sketches / wireframes of the interface before building 
the actual prototype. The resulting functional prototype was developed on a tablet with a touch interface. It was 
tested in a HITL study during January, 2013. 

The primary goals of the study were to collect feedback on the acceptability of the OASIS solutions and 
evaluate the interface. The results were quite promising. The OASIS advisory solutions were quite good and did not 
need much improvement. In fact, they generated solutions that were at least as good as the ones that the users 
generated themselves without OASIS. Participants also gave very positive feedback on the user interface but also 
suggested various enhancements to the OASIS tool which is planned to be incorporated into the next tool 
development cycle. Once the new design is completed, the OASIS tool needs to be tested in more realistic, dynamic 
traffic environments requiring greater tool interaction. A higher fidelity HITL simulation involving supervisors, 
controllers, and traffic managers or a field test trial would greatly benefit the understanding of how OASIS could be 
used to improve air traffic operations. 
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Appendix A: Development of Use Case for the Mid-Shift Change 


Trigger Description 

Plan and execute sector configuration change for mid-shift. 

Scenario: A short, specific 
summary of what is going to 
happen in this use case. 

At the beginning of the mid-shift when the traffic volume goes down or at the end of the 
mid-shift when the traffic volume goes back up, FLM plans and executes sector 
combinations to minimize the number of controllers when the traffic is reduced and sector 
splitting when the traffic volume increases. 

Triggering Event: The first user 
action in the use case; the event 
that starts the scenario. 

1 - 2 hours before the start of the mid-shift, FLM prepares for sector combinations that 
match the reduction in traffic volume. At the end of the swing-shift, FLM loses 4 - 5 
controllers to the end of their shift. A couple of swing-shift controllers remain until a 
replacement mid-shift controllers arrive to take their places. About an hour before the end 
of the mid-shift, a new FLM arrives with a couple of day-shift controllers. The day-shift 
controllers replace the corresponding mid-shift controllers. More controllers show up 
soon after as the traffic ramps up and new sectors are opened. FLM figures out which 
sectors to open first and prepares for sector splits. 

Actor: The user who is directly 
interacting with the interface; 
actually doing input on the 
device. 

FLM 
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Stakeholders: What users are 
interested in the results of this 
use case and what role do they 
play in it? Constrain the 
stakeholders tightly to the 
scenario. 

FLM coordinates with R and D controllers to execute the plan. FLM communicates with 
TMU about his plan of sector configuration change. FLM also coordinate with other FLMs 
about his plan. 

Can the tool give this information to other positions? Build in the data coordination 
mechanism in the OASIS tool? 

Pre-Condition: What needs to be 
in place before this use case can 
start? 

Staffing schedule is set and the new controllers for the shift change are arriving. Predicted 
traffic shows the traffic volume for the next 2 hours. FLM has checked with TMU about 
known TFM related issues - e.g. weather, wind-optimal routes filed by the airlines, etc. 

Post-Condition: How will the 
system have changed as a result 
of this use case? 

At the beginning of the mid-shift, sectors are combined with appropriate number of R and 
D-sides with the help of the OASIS tool recommendations. The tool may provide value in 
suggesting when to make the configuration change and which sector combinations to 
consider. At the end of the mid-shift, sectors are split back. The OASIS tool may provide 
value by allowing the FLM to delay opening new sector positions until the traffic ramps up 
and the new position is needed instead of opening the positions as soon as possible. 

Flow of Events: A short, specific 
story in chronological order. 

Look below that describes "Flow of Events: Step-by-Step" 

Errors: A list of things that could 
go wrong and optionally how the 
system might respond. 

El. The staffing schedule might change. 

E2. The actual traffic may differ from the predicted traffic. 

E3. The predicted traffic can suddenly change due to TMIs put in place. 

E4. FLM might find out about a TMI that will change the future traffic but is not reflected 
in the predicted traffic calculations. 

E5. Physical resources are not available (e.g. non-functioning scope, communication 
channels, radar outage, etc.) 


Flow of Events: Step-by-step 


Action Item 

Step 

Preplanned (Start of Shift or > 1 hour) 

FLM gathers information 

SI 

FLM gathers information - e.g. checks the TSD for the predicted traffic, checks Cru-ART for 
controller schedule, talks to TMU about the traffic and weather situation for the night. 

FLM checks the OASIS 
parameters 

S2 

FLM checks the OASIS parameters - i.e. checks to see that it has correct max/min values 
for the number of R and D-side controllers, correct max/min values for the number of 
sectors, correct max traffic threshold values per sector (optional), etc. 

FLM has a staffing plan in 
mind 

S3 

FLM has a sector combination/split plan in mind - i.e. which sectors to combine, in which 
order, where to put R and D controllers, etc. 

FLM goes to the OASIS 
tool and queries 

S4 

FLM goes to the OASIS tool and queries (push a button to re-compute) what its 
recommendation is for the sector configuration, timing of the configuration change, and 
the R/D assignments. The tool also gives FLM a high level summary of: 1) peak traffic at 
which sector, 2) min traffic at which sector, and 3) number of controllers needed. 

(optional) FLM "drills 
down" 

S5 

(optional) FLM "drills down" to view the predicted traffic situation, the staffing schedule, 
and the max traffic threshold per sector that the OASIS tool "sees" that contributed to its 
recommendation. 

(optional) FLM puts in the 
plan that he envisioned 
into OASIS 

S6 

(optional) FLM puts in the plan that he envisioned into OASIS to see what its impact is to 
the traffic. The OASIS gives back the summary results (i.e. peak traffic, min traffic, number 
of controllers). 

(optional) FLM modifies 
the OASIS 
recommendation 

S7 

(optional) FLM modifies the OASIS recommendation - e.g. change one of the sector 
combinations, move the D-side assignments to another sector, etc. FLM asks the OASIS 
tool to recalculate the recommendation based on the modification. The tool gives high 
level summary results. 
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FLM pushes a button to 
"activate" the plan 

S8 

FLM pushes a button to "activate" the plan - i.e. let the OASIS tool know that he will 
execute this plan. The tool takes a "snapshot" of the plan - i.e. sector configuration, 
timing, R/D assignments, traffic situation that the plan was based on, etc. From this point 
on, FLM can query this plan whenever he wants to jog his memory. What are metrics that 
needs to compare between the plan and the updated calculations based on current 
information (We need this for the OASIS tool to figure out when to alert the FLM of 
potential changes. FLM can also use it to keep track of the "goodness" of the original 
plan). 

FLM Coordination 

S9 

FLM coordinates with the controllers about his plan. 

OASIS Alerts 

S10 

FLM monitors for an appropriate time to make the configuration change. The OASIS tool 
may alert the FLM for a reminder at an appropriate time (if implementable). The OASIS 
tool may display the upcoming configuration and a countdown timer on the projector 
screen (if implementable). 

FLM executes the 
reconfiguration 

Sll 

FLM executes the configuration change at an appropriate time - e.g. put controllers on 
position, make configuration change in the HOST, etc. 


Appendix B: Usability Survey Results 


Question 

Average 

CRP: Plan View: Configuration Plan Graph 

Path 

Beta 

Was the information on this screen understandable? 

4.8 

4.3 

Was the information on this screen helpful? 

4.5 

4.0 

Were the colors helpful in displaying the information? 

4.5 

4.8 

CRP: Plan View: Sector Area Load Graph 

Was the information on this screen understandable? 

4.5 

4.3 

Was the information on this screen helpful? 

4.5 

3.8 

Were the colors helpful in displaying the information? 

4.8 

4.8 

Was the ability to highlight a sector or sectors in the line graph helpful? 

4.5 

4.8 

CRP: NAS Monitor View: Area Configuration Table 

Was the information on this screen understandable? 

5.0 

5.0 

Was the information on this screen helpful? 

4.8 

5.0 

CRP: NAS Monitor View: NAS Monitor Table 

Was the information on this screen understandable? 

4.5 

5.0 

Was the information on this screen helpful? 

4.8 

4.5 

Were the colors helpful in displaying the information? 

5.0 

4.8 

CRP: NAS Monitor View: Timeline 

Was the information on this screen understandable? 

4.8 

5.0 

Was the information on this screen helpful? 

4.8 

5.0 

How easy was it to select configuration change points using the Timeline? 

4.5 

4.3 

Was it easy to understand which time segment you were choosing on the Timeline? 

4.8 

4.8 

Were the colors helpful in displaying the information? 

4.8 

4.3 

CRP: Traffic Load Graphs View 

Was the information on this screen understandable? 

5.0 

4.8 

Was the information on this screen helpful? 

4.5 

4.5 

Was it easy to understand which time segment you were choosing on the Timeline? 

4.8 

4.5 

UGP: NAS Monitor View: Area Configuration Table 

How easy was it to make changes to the airspace configuration using this table? 

5.0 

5.0 

How easy was it to make changes to the D-side assignments using this table? 

5.0 

5.0 

UGP: NAS Monitor View: Timeline 

How easy was it to add, copy, move, and delete configuration changes using the Timeline? 

4.8 

4.8 

General Questions 

How easy was it to navigate in OASIS using the "Mode" and "Action" buttons along the bottom of the screens? 

4.8 

5.0 

Were you able to easily distinguish between the "Active Plan," "Computer Recommended Plan," and "User Generated Plan" modes? 

5.0 

4.5 

Overall Average 

4.7 

4.6 
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A. Plan View 

• Overall comments: used screen often, found hard to decipher, least used screen 

• Suggestions', sectors that are routinely combined be placed next to each other, allow facilities to set the 
order in which sectors are listed, show total staffing number, could we do more or everything on the Plan 
View screen? 

B. Sector Area Load Graph 

• Overall comments: a lot of data, useful for seeing traffic load, took a while to figure out 

• Suggestions: show single sector’s traffic, not just combined sectors’ traffic; use green as a highlight color, 
use the legend to select graph lines 

C. Area Configuration Table 

• Overall comments: several liked this screen, allowed experimentation, easy to read, well done 

• Suggestions’, stars indicated changes, but sometimes I could not remember what change had been made 

D. NAS Monitor Table 

• Overall comments: well designed, used frequently, easy to understand, two did not use it much 

• Suggestions’, show a two-hour look ahead and move data shown on table to the left as time progresses 

E. Timeline 

• Overall comments: liked it, used it all the time 

• Suggestions’, selecting a time segment on the NAS Monitor view should carry over to the other views, 
difficult to select because of tablet cover and small size of Timeline, might only plan 45 min to 1 hour 
ahead 

F. Traffic Load Graphs 

• Overall comments: most helpful tool, did not refer to it very often, easiest to see sectors that needed 
attention, never used it, good display of traffic peaks and valleys 

G. UGP Area Configuration Table 

• Overall comments: very easy to use and understand, crashed some but liked it, manipulating sectors was 
easy, user friendly 

• Suggestions’, should be able to operate a sector at a different physical position, drag D-side to trash can to 
delete, do all this page’s activities from Plan View 

H. UGP Timeline 

• Overall comments: not too comfortable with this at first, but got better with practice; easy enough to use, 
using fingers on timeline was not easy at first, simple to use 

• Suggestions’, drag the time of a configuration to another time spot, without cut and paste 

I. General Questions 

• Overall comments :: used all screens, found second and third to be most useful; used third screen only a 
little, plan view was more difficult to use so used it less; used the first screen the least 

• Suggestions’, a help feature would be useful, especially for new users 
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